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(57) Abstract 

A method and a 
device for displaying at 
least part of a person's 
body, for example 
the head (10), with a 
modified appearance by 
providing first image 
data rqjiesentatitve of the 
unmodified appearance 
of the body part, second 
image data repnesentatitve 
of a modification in the 
appearance of the body 
part (10) and position 
data representaUve of 
the position and spatial 
orientation of the body 
part (10). The first and 
second image data are 
linked to produce and 
to display third image 
data representative of 
the modified appearance 
of the body part, in 
dependence on the 
position data, such that 




AL 

AM 

AT 

AU 

A2 

BA 

BB 

BE 

BF 

BG 

BJ 

BR 

BY 

CA 

CF 

CG 

CH 

CI 

CM 

CN 

CU 

cz 

D£ 
DK 
EE 



Codes used to identify 

Albania 

Annenia 

Austria 

Australia 

Azerbaijan 

Bosnia and Herzegovina 

Barbados 

Belgium 

Burkina Faso 

Bulgaria 

Benin 

Brazil 

Belarus 

Canada 

Central African Republic 

Congo 

Switzerland 

C6ie d'lvoiie 

Cameroon 

China 

Cuba 

Czech Republic 
Germany 
Denmark 
Estonia 



FOR THE PURPOSES OF mPORMATION ONLY 
States party to the PCX on the fiont pages of pamphlets publishing 



ES 


Spain 


FI 


Finland 


FR 


France 


OA 


Gabon 


GB 


United Kingdom 


GE 


Georgia 


GH 


Ghana 


GN 


Guinea 


GR 


Greece 


HU 


Hungaiy 


IE 


Ireland 


IL 


Israel 


IS 


Iceland 


IT 


Italy 


JP 


Japan 


KE 


Kenya 


KG 


Kyrgyzstan 


KP 


Democratic People's 


KR 


Republic of Korea 


Republic of Korea 


KZ 


Kazakstan 


LC 


Saint. Lucia 


U 


Liechtenstein 


LK 


Sri Lanka 


LR 


Liberia 



LS Lesotho 

LT Lithuania 

LU Loxembomg 

LV Latvia 

MC Monaco 

MD Republic of Moldova 

MG Madagascar 

MK The former Yugoslav 

Republic of Macedonia 

ML Mali 



MN 


Mongolia 


MR 


Mauritania 


MW 


Malawi 


MX 


Mexico 


NE 


hfiger 


NL 


Netherlands 


NO 


Norway 


i\Z 


New Zealand 


PL 


Poland 


PT 


Portugal 


RO 


Romania 


RU 


Russian Federation 


SD 


Sudan 


5E 


Sweden 


SG 


Singapore 



international applications under the 

SI Slovenia 

SK Slovakia 

SN Senegal 

SZ Swaziland 

TD C3iad 

TG Togo 

TJ Tajikistan 

TM Ttnkmenistan 

TR Turkey 

TT Trinidad and Tobago 

UA Ukraine 

UG Uganda 

US United States of America 

UZ Uzbekistan 

VN Viet Nam 

VU Yugoslavia 

ZW Zimbabwe 



wo 99/23609 



PCT/NL98/00625 



A method and a device for displaying at least part of 
the human body with a modified appearance thereof. 

The invention relates to a method and a device for 
displaying at least part of a person's body with a modified appearance 
by providing first image data representative of the unmodified appearance 
of said body part, second image data representative of a modification in 
the appearance of said body part and position data representative of the 
position and spatial orientation of said body part, wherein said first 
and second image data are linked to produce and to display third image 
data representative of said modified appearance of said body part in 
dependence on said position data, such that said displayed modified 
appearance follows changes in the position of said body part in a dynamic 
manner. 

Many clients of hairdressers, beauticians, jewellers, 
dentists, opticians or boutiques have a need for support and advice in 
the selection of their purchases or treatment. A person's imaginative 
powers continue to play an Important role in this respect. Not everybody 
possesses the same talents in this regard and consequently it is attractive 
to be more sure of the result in advance by being given an adequate 
personal visualisation thereof. This applies in particular in those cases 
where the result cannot be immediately undone, as may for example be the 
case with a hairstyle. The decision to take a particular treatment and 
the selection of a particular type of treatment can be made more 
efficiently when said decisions are supported by such a visualisation, 
as a result of which the time involved in said treatment can be reduced.' 

US Patent No. 4,823,285 discloses a method for 
representing a person with a modified hairstyle by means of a computer, 
a television camera and a viewing screen. A static video image of the 
respective person's head is made, which is displayed on the screen. After 
a new hairstyle has been selected from a data file, the original hairstyle 
is substituted on the screen for the selected new hairstyle. 

US Patent No. 5,404,426 discloses a method and a device 
for visualising hairstyles, using graphic computer techniques. Unlike the 
presentation of static hairstyles as disclosed in the aforesaid US Patent 
No. 4,823,285, a system is disclosed by means of which the hairdo can be 
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modified interactively on the screen. To this end a computing model is 
used, by means of which modifications in an artificially formed hairstyle 
resulting from external forces are represented. Said external forces, among 
which the influence of gravity or the forces resulting from the modelling 
of a hairdo by means of a comb or the like, can be simulated via a computer 
mouse or a computer keyboard. 

It has become apparent that such a visualisation will 
only have the intended supporting effect in selecting a purchase or a 
treatment if said visualisation can appeal to a person's intuitive 
behaviour, wherein perception plays an important role. Said perception 
is considerably enhanced by movement. 

US Patent No. 4,539,585 discloses a prevlewer for the 
dynamic viewing of article desired to be worn by a user in a complete 
series of poses without actually trying on the article. In an embodiment 
the user creates in real time a series of poses which are time-framed and 
position orientation coded. The image of the article is computer generated 
and re-imaged in accordance with the coded position information of the 
user. The re-Imaged article image and the user. video image are than video 
mixed, optically mixed or superimposed in digital image storage memory, 
such that a composite Image may be viewed by the user in a dynamic viewing 
mode. 

Unlike the above-described prior art. this prevlewer 
makes it possible to present the modified appearance of a user in a 
dynamical manner, such that the person in question can evaluate the effect 
of the modified appearance in several positions thereof. However, because 
the position coding depends on coded indicia segments, which segments in 
toto form a composite image of the article, only a limited number of poses 
can be actually covered. 

Although this known prevlewer appeals more to a person's 
Intuitive behaviour than the static presentations, in practice, there is 
a need for a more sophisticated presentation technique such that a modified 
appearance of at least part of the human body can be displayed in a true- 
to-life manner, as if the person is looking at himself In a mirror with 
the modified body part. 

Accordingly, it is an object of the invention to provide 
an improved technique for displaying a body part with a modified 
appearance, which enables the person in question to evaluate the effect 



PCT/NL98/00625 

3 

of the modified appearance in a more true-to-life manner. 

According to the invention for linking said first and 
second image data a virtual image data space is created comprising an 
aspect or aspects of said appearance of said body part to be modified, 
wherein said second image data comprise virtual image data elements, and 
wherein said modification of said appearance of said body part is provided 
by modifying said virtual image data space using virtual image data 
elements of said second image data, wherein said virtual image data space 
thus formed and said first image data not comprised thereof are linked 
to produce said third image data in dependence on said position data. 

The term virtual image data space is to be construed 
in that a real image data space of the first image data representative 
of the unmodified appearance of the body part and comprising the aspect 
or aspects of the body part to be modified, is virtually reconstructed. 
Next, the actual modification is carried out by modifying the virtual image 
data space using second image data comprised of virtual image data 
representative of the intended modification. The modified appearance is 
than created by suitable linking of the modified virtual image data space 
and the first image data, providing third image data representing the 
modified appearence of the body part. 

The virtual image data space concept according to the 
invention is essentially able to cope with any changes. in the position 
and spatial orientation of the body part with respect to its environment, 
such as shading due to differences in the incident light, hiding of 
portions of the environment due to movement of the body part, etc. This, 
because the influences of such changes on the appearance to be displayed 
are virtually constricted in the virtual image data space comprising the 
modified aspect or aspects of the appearance using respective virtual image 
data elements. 

Unlike the above-described prior art, the invention 
makes it possible to present the modified appearance in a more dynamical 
and true-to-life manner, such that the person in question can evaluate 
the effect of the modified appearance in essentially unrestricted positions 
thereof. It will be appreciated that such a dynamic, interactive 
representation is a more efficient support of a person's imagination, as 
a result of a stronger true-to-life perception. That is, the modified 
appearance directly follows the changes in the position of the body part 
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in a true-to-life manner, as if the person is looking at it in a front 
mirror with the body part thus modified. 

The method according to the invention provides to select 
patterns or models in connection with a desired modification, which 
patterns or models may be selected from a data file, if desired. The actual 
modified appearance is than created in accordance with the patterns or 
models selected and composed of virtual image data elements. 

Selection of a respective pattern or model may take 
place in several manners, for example by making a selection from a pattern 
or model book, whether or not electronically, whereby the appearance of 
a respective body part is shown commensurately with a respective pattern 
or model , 

The selection of a particular pattern or model may be 
provided in an efficient manner by first displaying, in accordance with 
another embodiment of the method according to the invention, the respective 
part of the person's body in a fixed position with a modified appearance 
commensurate with a pattern or model to be selected. That is, the body 
part in question is displayed in a particular position, such as a reference 
position, with a modified appearance, so that the person in question is 
given a first indication of the influence of a pattern or model on the 
respective part of his own body. The patterns or models that have been 
selected in this manner can then be used for visualisation in a dynamic 
and true-to-life manner, as described above. 

It has furthermore become apparent that the selection 
of a particular pattern or model can also be supported by displaying fur- 
ther information in combination with the modified appearance, which further 
information may be visual information or auditive information, or both. 

In this connection, further information directly 
relating to the modified appearance, such as information with regard to 
care, cost, possible risks and the like may be considered, but for example 
also information which places the person in question with his modified 
appearance in a desired situation, a decor, for example, but also clothes 
and the like. In accordance with one embodiment of the method according 
to the invention, said further information may be provided in the form 
of (moving) images, such as a video clip. In this way it is also possible 
to initiate exchange of information by means of questions and/or answers. 
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Besides the representation of the modified appearance 
commensurate with a selected pattern or model, the invention, in a 
preferred embodiment thereof, furthermore provides a possibility to modify 
one or more aspects of the modified appearance interactively during said 
representation. A modification in the colour, the volume, the length and 
the like can be considered. 

The technique according to the invention is not limited 
to one or more specific parts of a person's body, actually, any desired 
body modification can be simulated in a manner which optimally matches 
a person's perception, so as to support said person's selection of a 
respective modification. 

The invention is especially applicable as regards a 
person's head, wherein the modified appearance comprises modifications 
in one or more aspects of the hairstyle, the make-up. accessories such 
as glasses, coloured contact lenses, earrings, piercings and the like, 
and also modifications of the nose, mouth, lips, teeth, eyes ears' 
eyebrows, moustache, beard and other facial features, among which the 
effect of a i^acelift or other reconstructive work, for example, wherein 
for each aspect to be modified a separate virtual image data space is 
created or wherein aspects to be modified are comprised by a common virtual 
image data space. The modifications as such, i.e. the hair. nose, etc 
are formed from respective virtual image data elements composed of 
polygons, texture etc., for example. 

The first and the second image data can be linked in 
various manners, according to an embodiment of the method of the invention 
for example, this is provided by first arranging the second image data 
in a relative position commensurate with the first image data. The second 
image data can be selected from a plurality of pre-stored virtual image 
data elements in a data file, or be computed by means of suitable 
mathematical algorithms, on the basis of the position data as address 
Information. 

In the case of a selection from pre-stored second image 
data comprising data recorded from various angles and positions, those 
image data are selected therefrom during representation that correspond 
most closely as regards the recording angle or position with the actual 
angle or position of the respective body part. Next, the image data for 
the exact angle and position can then be derived by means of an 
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interpolation method. Mathematical techniques for computing such views 
comprise, inter alia, descriptions including one or more characteristic 
parameters and polygons. 

In a preferred embodiment of the method according to 
the invention, the virtual image data space is created relative to a 
generic type of said body part, wherein data representative of the 
dimensions and geometry of said body part are generated from said first 
image data, on the. basis of which data said generic body type is scaled 
into commensuration with said dimensions and geometry of said body part 
according to said first image data. 

The advantage of this manner is that the virtual data 
space can be directly generated from the correct viewing direction, as 
a result of which it can be adapted relatively easily and fast to'the 
movements of the body part. 

In a yet further emobidment of the invention, the second 
image data are scaled into commensuration with said scaled generic body 
type, and said virtual image data space is modified by said scaled second 
image data. 

The image data are linked in such a manner that portions 
of the respective body part which will not be visible in the modified 
appearance are not shown. For example, assuming that the person in question 
wears glasses and wishes to see how he looks with a new frame, the data 
relating to the current glasses of the person in question will have to 
be removed from the first image data. It should be considered thereby that 
the person in question will necessarily have to use his glasses in order 
to be able to view the modified appearance. 

Another problem which plays a role in this connection 
is known as occlusion. When the first and the second image data are being 
linked, it is taken into account that details of the body part in question, 
which in reality are hidden behind a selected pattern or model, and which 
are not visible, therefore, must not be shown in the modified appearance 
being displayed, either. 

On the other hand, a particular modification, for 
example a different hairstyle, may make it necessary to show parts of a 
person's face which are not provided by the image data which are 
representative of the unmodified appearance of that person. In such a case, 
extrapolation of image data is required by adding respective virtual Image 
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data elements. Such as derived from the generic body type ' 

by makina th. . . '"""^^"^ to the invention, 

sir:;:: r r ^-^ 

so-called 'tr.ckers., which provide position data relating to the post«o„ 
and the spatial orientation (angle, of the bod. part in ,Lst „ I ^ 
appreciated that according to the invention said tracking ^ans L 
as they are visible in the first i.age data. n,ust not be visible „ the 
third image data being displayed. "sioie in the 

-esentation or slllar of^heTried^rr'^he' iirrt" 

;::rcrof .;;:r -r "^'"^^ - ^'^^ -r;;tr„ l:: 

rrequency of 25 pictures/second or higher. 

Unlike the prior art discussed above the virtu;,! in,=.„= 
ata according to the invention provides for three-di.en;i!„a proces ^ 
where,„ the modified appearance can be displayed both t„o-d,. „s 
P ov d iTr: ■ P-"sing is very advantag f„ 

ZZZl ""^""-"'^ '° 'He present 

at invention also provides a device for displaying 

at least one part of a person's body with a edified appearance thereof 
as discussed above, which device comprises first ^eans for prov d „g rst 
.age data representative of the „n™dified appearance of the body p 
second means for providing second i^age data representative of 
^od,f, cation ,n the appearance of the body part, position determining means 
for providing position data representative of the position and spatial 
orientation of the body part, processing means for electro ca,y 
processing the first and second image data in dependence on the 0": 

0 th b d ' '"^ -P-sentative of the modified appeara ce 

Of the body part, and means for displaying the third image data 
haracterized in that the second means are arranged for providing second 
«age data having the fo™ of virtual image data elements the proc s g 

aspect Of the body part to be modified, for adding to the virtual image 
data space virtual image data elements and for linking the virtual lage 
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data space thus formed and the first imaae d;*t;. • . 

^ I ir:>i. image data not comprised by the imanp 

"purr:."""" ^"-^ - - -"-^-^ 

the • ^" ^ preferred embodiment of the device accordinq to 

er;: — e . least one fixedl. dJZed 

vi^eo camera, and the tracking means comprise one or more trackers for 

body part relative to the recording means. respective 

disnn.pH ^ " ""^"^ '^"^"^ "'"^''^ also be movably 

disposed for the purpose of the invention. Furthermore several vidio 
cameras may be used for recording, and displaying the body pt, 

p~ '\ r""* ^"^'^"^^^ three-di^: i 

processing and relating to the generic type body part disclosed above 

The invention in particular provides a device comori^inn 

d posed on t -7"''""' ^'^"'^^ 

s de oTlV , ' '•"P"""^ other 

side of the semi-transparent mirror. 

. ^ according to the invention which is 

configured ,n this manner, the person in question looks in a mirror s 

Of t e actual body part. Let us assume a situation wherein the body part 
>n , est,on ,s a person's face. In that case the recording means and the 
d spiay means „i„ be arranged in such a manner with respect to th sem - 
transparent mirror, that said person directly looics in th record me" s 
a th m,rror. This creates an ,„usio„ of the person in question io^ „g 
in a real m,rror. and all the movements he makes in front of the mirror 

— « 

the invention, means' 11 Z:rdrf:n:::: d^ta"; " 

storing data comprising one or more 

tterns or models which are representative of an intended modificat o 
d d r";r" °' ' '""^ """tion may be 

( b.tmap ) or data which can be processed into video data, like virtual 
image data elements. "rtual 
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' manner which is known per se ' ^" ' 

data. The transmission means may for example consist of a suiLhiv 

data a„d/„r ,„age ,de„t*ficatt„„ o. selection data are , inked to the ]lZ 
• ^" another embodiment of the device accordina tn 

nnect.on, the use of a computer mouse, a keyboard, touch screen and th. 
'>ke for „od,f„„g the colour, the vo1ua,e and th Hke Tf , ld,f . 
appearance nay he considered. ,„ an advantageous e J l„t thi! „ ^ 1' 
's controlled intuitively, in such a manner that a Ir' hav ino s 1 
experience in the control of computer systems can op e h 'de Z 
himself without constant guidance. 

^ X ^""^ ""^^ according to the invention is furthermore 

arr nge for storing a selected modified appearance by providing a so 
caned hard copy, for example, by means of a (video, printer r a soft 
opy on a diskette, a smart card or other informatl „ car or u t b, 

nlrali ' ^-'"'^ -c l tt 

information, such as co„erciaI tnfonnation or infomation on costs care 
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etc. 

be provided with J!nVf '° ^-thermore 

. -I.e. appearance .ein. disp.a.ed, asX:;;;" ■ ^ .Xr" 

b. .eans o. an e.e J";:" ^ ^ --ter 

according to the Invlntion "'^^^'^"^^^ ^^^^ ^ ^^^^^-nt of the device 
9 the invention, comprising a semi-transparent mirror 

Video data accord ing'tTt^: ;r:r"^ ''''' 
the invention by me^nfrf lich"'';""' 

be Visible are mask d in th. ' °' ' '''' '''' ^^'^ 

masked in the representation with the modified appearance 

Hgure 4 schematically shows in a hi^^-i. a- 
thereof. ' ° ^"^ ^"^^^ "P ^n embodiment 

rr;.~rTH™~— ^^^^^^^^ 

rdce, such as modifications in the hairstvle mako i 

earrings, etc., or ™diflc.tio„s of the nose ^"th" T^T ' 

eyebrows, moustache, beard and the like It shoLlTh . V 

that ffc. J Should be understood, however 

that the invention is not limited thereto. however. 

head is to h. 1 ' ''8"" 1 indicates a person whose 

IS to be displayed with a modified hairstyle as the new appearance 

;7-„ substanti:;;:ir:T;em:;ra::a" n:":r^ 
sir:: tT^nTwh? °"°"r - - thre::!!.::: 

of theLr, ! ' ' "'^P-^^" °" tl>= other side 

Of the semi-transparent mirror. The semi-transparent mirror 2. the disp ay 

the representatlonThl! Z T::^:::, ^ ' """'^ 
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L 3 ' i,? "'""'^ ' ""P"ter. A„ output of 

tracMng .ea„s 6 are connected „ a T / „ 7T 

5. ^ Turxner input of the processing unit 

"eans 7, for exa™,JT„ tTr'"';"' ' '"'"''™°" """'^'^ *° 
' storing d„na?Tata " ' '^'^ "'^ "^^^^ 

co,»orc1ally avallabuf"'" according to Figure I ™ay be assembled from 
the inven on Process :rmr"r ■ 

- components io^e^t^^ ^f ^ ZT"' 

:: ~ ~ 

supDlied to th= """'^ "olivered by the video camera 4 are 

aitr.^ ai, r ^ - 

data h„ ■ converted into digital video 

data by the processing means 5 using a so-called "frame grabber" 

to be modified one l7 ZTnJ T"' — 
present example part of th. n . Processing means 5. In the 

»^ - arc::p:::odT: :ir;::::i:r: 

Second image data from the data file 7 wh.VK 



wo 99/23609 

PCT/NL98/0062S 

12 

Next, the modified virtual image data space is linked 
or merged with the remainder of the first image data to provide a complete 
modified appearance. The thus formed third image data may be supplied to 
the display means in digital form or in analog form, depending on the type 
being used. It is preferred to use digital data exchange. 

The third video data produce an image 8 on the display 
means 3, which is a combination of the unmodified view of the face of 
person 1, i.e. real first image data and the modification effected therein 
by means of the virtual second video data from the storage means 7 The 
image 8 can be viewed by the person 1 via the semi-transparent mirror 2 
The dimensions of the image 8 are thereby such that person 1 perceives 
what he sees through the semi-transparent mirror 2 as a representation 
of his own head with the new hairstyle in question. 

Movements of the head are transmitted to the processing 
means 5 via the tracking means 6. On the basis of the position data being" 
delivered, the modified virtual image data space is processed and linked 
with the first video data from the video camera 4 to produce third video 
data, such that it seems to person 1 as if he or she is looking at himself 
or herself in a mirror. By using a video frequency of 25 pictures/second 
or higher, the movements of the head can be followed dynamically without 
an objectionable time delay, so that also the modified appearance 8 of 
the respective body part will be displayed dynamically and in real time 
for user 1. 

In this aspect, the present invention essentially 
differs from the prior art. which is only capable of displaying dynamics 
of a respective body part in a limited fashion. The virtual image data 
space concept of the invention enhances the value of the perception, which 
makes it possible to take a decision as regards a particular type of 
treatment and the execution thereof in a more effective manner. 

Figure 2 schematically illustrates the linking process 
of the first image data and the second image data in an embodiment 
according to the invention. 

Reference numeral 10 shows a person's face with real 
hair 11, i.e. first real image data. 

From the first image data, parameters representative 
of the dimensions and geometry of the body part, i.e. the face 10. are 
derived by the processing means 5. The dimensions and geometry obtained 
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Z\Z\l: *° ' '"^"-^^ ""-^ ^ =-Hc face 

„< , , . ^'"^ generic body part 12 is used to create a 

virtual mage data space of the aspect or asoects of th. 
bemodifiert ( « ft k ■ . , aspects of the appearance to 

be nod f ed. ,.e. the hairstyle 11. ,„ the figure, a possible virtual Image ' 

dins ^1 " . ' «1*'-9h Shown t„o 

dl»e„s,„„ally, virtual 1„age data space , preferebaly Is three-d1™ens1„n" 

1, h ^. » '■'"^'■■"'^"^'"'''"We 13, which Is a virtual Imaoe 

IS brought to scale using the scaling parameters of the generic bodyZ; 

The second image data from the storage means 7 which 
are representative of the hairstyle 13 and selected by a person 1 re 
added to the virtual data space 9 and linked with the first t ; the 
«age 12 to produce third Image data, in such a manner that a v rtu , 

rtrirhr^triirr^--^^^-- " '-^ - 
c..era . for e. Jet ::,:^:r:z:::sr ^^Jcr!: 

The virtual data space 9 ,s modified again In accordance with the changed 
po tion, es well Is the hairstyle ,3 corresponding with the respec ve 

the c " T'"'"' ' '-^^ ' corresponding w t 

linking of the respective data (see Figure 1). 

which generally represent a 
discrete positions, whereby the position data delivered by the tracking 

An optmu,, between positions can be found by means of interpolation 

from thp ""^'"^ of = respective pattern or model 

from the storage means 7 on the basis of the position data, the pattern 
or mode data corresponding with a particular position or oHenta o 
the head or another body part may also be computed on the basis of the 

example, a hairstyle or other pattern or model with associated parameUrs. 
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Unlike the pattern or the model itself, stripped of 
all non-relevant environmental influences, this may also be stored in a 
data file in relation to a generic standard type of the respective body 
part. Based on the first image data of the body part of a person in 
question, a scaling of the generic, three-dimensional type of body part 
to fit the body part of the person in question may take place in that case 
Then the above-described procedure may be followed again for dynamically 
displaying the modified appearance in accordance with the momentaneous 
movements of a person. 

The first image data of the person's face and the second 
image data relating to the hairstyle that has been selected or that is 
to be selected are aligned in advance by linking the actual position of 
the person's head once to the read-out of the tracking means 6 (Figure 
1) in a coordinate system to be selected. The person may take up a 
reference position for said calibration, for example looking straight in 
the mirror 2, head upright, etc. 

According to the invention it is possible in such a 
reference position to offer a person a first selection possibility for 
selecting a desired pattern or model for subsequent dynamic visualisation. 

When modifying the virtual image data space by the 
second image data, the fact that details of the pattern or model and of 
the body part in question will not be visible in practice, because they 
are visually masked, is taken into account, of course. In an embodiment 
of the invention, these parts are determined on the basis of a three- 
dimensional, standard type of the body part. The third image data to be 
displayed are adapted accordingly in that case. Also in this case it 
applies that the generic, three-dimensional model of a respective body 
part is adapted again to the dimensions of the real body part in order 
to analyse to scale the details that must not be displayed. This analysis 
is effectively enhanced by providing the generic body type part in a 
transparent mode. Picture analysis techniques which are known for this 
purpose are well-known to those skilled in the art. In this connection 
reference is made inter alia to literature on Z-buffering. 

Figure 3 schematically shows a method for determining 
which portions must not be displayed, which portions may include the 
tracking means 6 (Figure 1), and spectacles, for example, if the modified 
appearance involves a new spectacle frame. 
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In Figure 3, the virtual hairstyle 13 is stripped of 
the original environment, for example a face and a background. The hatched 
portion 14 is a transparent generic model of the face, which is used for 
determining which portions of the eventual face to be displayed must be 
visible. In the illustrated view, for example, part of the right-hand side 
of the face (seen from the drawing) is hidden from view by the hair 13. . 
Techniques of this kind are known by the term occlusion in practice. 

On the other hand it may be necessary to generate, by 
extrapolation, portions of for example the face which are not visible in 
the first image data, because they are shielded by the existing hair, for 
example, but which must be visible in the third image data. 

Besides the dynamic adaptation of the modified 
appearance on the basis of a person's position and movements, it will be 
apparent that hair, for example, is not a rigid object, but makes a 
specific movement of its own under the influence of the person's movements. 
One embodiment of the Invention, in particular intended for displaying 
hairstyles, distinguishes those parts of the hair that react differently 
to forces caused by movement, for example by throwing the hair back, by 
gravity and by movements of the air. The eventual hair movement that 
results is realised by computing a resulting acceleration from the 
respective natural movement, the position of the head, movement of the 
air, gravity and parameters relating to the hair itself. In the last- 
mentioned case, the difference between long thin hair, long thick hair, 
short hair, etc. may be considered. All this is done in order to provide 
a true-to-life. simulation. 

In the case of a hairstyle, for example, it will be 
apparent that operations may be carried out thereon, such as a treatment 
of the hair (structural change) in the form of conditioning, setting, 
curling, permanent and the like, or a partial or complete colour 
modification (colour modernisation). According to the invention, this is 
realised by offering the user a possibility to change one or more aspects 
of a respective pattern or model in an interactive manner while said 
pattern or model is being displayed. 

One or more so-called modification vectors can be 
defined for this purpose, which vectors relate to a particular part of 
a pattern or a model, such as a hairstyle. A suitable selection and/or 
grouping of modification vectors makes it possible for example to effect 
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a local or a total modification in volume, a local or a total modification 
in length, a local or a complete modification in colour, etc. It is 
possible to apply said modifications only to the second video data of the 
pattern or model, or directly to third video data being displayed, if 
desired. 

Colour modifications can for example be effected by 
modifying the value of colour parameters from the second image data in 
a selected part. This means that colours need not become uniform, but that 
differences in shape will be retained. 

Make-up can for example be applied in a comparable 
manner by selecting colours in a specified area, for example the lips, 
and modifying said colour. Thus it is possible to modify parts of a face 
that stand out. In those instances where no parts stand out, or not as 
desired (a blush on the cheeks, mouth enlargement), said parts will have 
to be caused to stand out in the third video data. In order to prevent ' 
undesirable modifications of the same colour in other parts of the image 
being displayed, the area within which the colour modification is allowed 
to take place can be delimited by means of a suitable marking technique. 
Only the selected area will be modified as well on the basis of the 
position data from the tracking means 6 when the body part, the head in 
this example, is moved. 

Accessories such as glasses, earrings, necklaces etc. 
can be generated from a mathematical description of the shape and the 
colour related to a three-dimensional standard type of body part. In 
practice, this technique is known as "Virtual Environment". 

Figure 4 shows in a block diagram the various electronic 
components and control means from which the device according to the 
invention can be assembled. 

The heart of the system consists of digital processing 
means 5, inter alia comprising a microprocessor, memory means for the 
storage of software, a main memory for interim data storage, a "frame 
grabber", a graphic card and the required interfaces and connections for 
data exchange, among which graphic and audiovisual interfaces and the 
required software. 

One or more cameras 4, one or more trackers 6 (tracking 
means), display means 3 which may be in the form of a viewing screen and 
loudspeakers, and storage means 7, according to what is desired connect 
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to the processing means 5. Second image data relating to patterns and 
models may be stored in the storage means 7. Further, if desired 
background information and other information to be displayed in combination 
with a modified appearance may be stored in the storage means 7. Further 
digital data may be stored instead of image data (bitmap), from which " 
further data the second image data can be generated. Furthermore a printer 
15, a computer mouse 16 or other control command element, and a keyboard 
17. as desired, will be connected to processing means 5. It is possible 
to provide a print of a body part with a modified appearance and possibly 
other information via printer 15. whilst computer mouse 15 and/or keyboard 
17 may be used for giving commands for interactively modifying one or more 
aspects in an image being displayed. To this end the display means may 
be of the touch screen type. Instead of a printer 15. pictures may be 
electronically provided and outputted to a user via the Internet for 
example. 

It is aimed at to make the control of the system as 
simple as possible, for example by means of a graphic control or a control 
menu. The storage means 7 may be present at the same location as the device 
according to the invention, but they may also be present remote therefrom 
and be connected thereto via a data communication link, for example via 
the public telephone network or a data network, the Internet or the like. 
The storage means 7 may also consist of detachable registration means 
such as a smart card, a diskette, a CD-ROM etc. All this in order to enable 
an optimally flexible and simple adaptation of software and patterns and/or 
models for the purpose of following trends and the like. 

The computer- readable storage means 7 provided with 
second image data or with further digital data from which second image 
data can be derived, and the manner of addressing and arranging the 
respective pattern or model data as described above form part of the 
invention. Instead of or in addition to the storage means 7, a model or 
pattern generator 19 may be provided for generating second virtual image 
data. Generic type model or body data may be stored in the storage means 
or separate storage means 20. 

The arrangement shown in Figure 4 may be built up of 
components which are available in practice, among which a modern multimedia 
personal computer having a processing speed of 120 MHz or higher and a 
working memory of at least 32 Mb. It will be apparent that the accuracy 
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of the modified appearance that Ts displayed largely depends on the 
processing capacity of the processing means 5 in question. It is noted 
that It IS not accuracy which is the main objective of the present 
invention, but rather the perception, in order to appeal maximally to a 
person's imagination. 

"Trackers" which are suitable for the purpose of the 
invention are commercially available, and they may for example operate 
according to the electromagnetic, the acoustic or the infrared principle. 

The user can operate the device according to the 
invention himself, without requiring a great deal of experience After 
a respective body part has been prepared, which may be necessary, for 
example hair being tied tightly backwards. In such a manner that It is 
possible to show as much as possible of the person's face, the tracking 
|neans 6 are attached. This may for example be done by means of a clip 
behind the ear. Then the person takes up a prescribed position in front 
of the camera 4 for the purpose of calibrating the tracking means. After 
a number of parameters which may be of relevance for the possible 
selections, for example the density of the person's own hair, the length 
the colour and other preferences have been input, displaying may be started 
at once, or a selection from an electronic catalogue of patterns or models 
may be made first. According to the invention, models from said catalogue 
can directly be shown on a representation of the person's face. Once the 
selection has been made, displaying Is started. 

. ^ ^" to enhance the perception, the side or the 

back of the head or another body part may be displayed on the display means 
3 in one or more windows thereof. Two or more video cameras may be used 
for this purpose. The images in the windows can be displayed so that they 
can be easily viewed by said person. 

The environment in which the persons sees himself is 
also of great importance for the perception. Consequently, a recording 
room may be provided, which comprises suitable lighting, background, sound 
and other elements which determine the atmosphere, and which can be 
adjusted in dependence of a respective selected model or pattern In the 
case of a hairdresser's salon, for example, the recording room may be a 
separate room or it may be integrated in the salon Itself. 

Fixed video cameras are preferred in order to keep the 
construction simple, the invention is not limited thereto, however. 
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Besides the processing means 5, input/output (I/O) means 
18 are furthermore provided for external control of for example elements 
which determine the atmosphere in the recording room, but also for relaying 
a recording to one or more remote display means 3. in such a manner that 
others can also view the recording. Said relaying may take place in a well- 
known manner, both via cables or by wireless means. 

For statistical and historical purposes, a user log 
file may be automatically created and analyzed by suitable software. 

Although the invention has been illustrated in particu- 
lar by means of an application for representing a person's face with a 
modified hairstyle in the foregoing, it stands to reason that the invention 
and Its uses are not limited thereto. 



wo 99/23609 
CLAIMS 



20 



PCT/NL98/00625 . 



^' A method of displaying at least part of a person's body 

with a modified appearance by providing first image data representative 
of the unmodified appearance of said body part, second image data 
representative of a modification in the appearance of said body part and 
position data representative of the position and spatial orientation of 
said body part, wherein said first and second image data are linked to 
produce and to display third image data representative of said modified 
appearance of said body part, in dependence on said position data, such 
that said displayed modified appearance follows changes in the position 
of said body part in a dynamic manner, characteri7ed in that for linking- 
said first and second image data a virtual image data space is created 
comprising an aspect or aspects of said appearance of said body part to 
be modified, wherein said second image data comprise virtual image data 
elements, and wherein said modification of said appearance of said body 
part is provided by modifying said virtual image data space using virtual 
image data elements of said second image data, wherein said virtual image, 
data space thus formed and said first image data not comprised thereof 
are linked to produce said third image data in dependence on said position 
data. 

^' A method according to claim 1, wherein the appearance 

of a respective body part is modified commensurately with a preselected 
pattern or model composed of virtual image data elements. 
2- A method according to claim 2, wherein a respective 

pattern or model is selected from a file of patterns or models. 

A method according to claim 2 or 3, wherein said body 
part is displayed with a modified appearance commensurate with a pattern 
or model in a reference position, in order to select a respective pattern 
or model . 

^' A method according to any of the preceding claims, 

wherein the appearance of said body part is modified interactively in one 
or more aspects during display thereof. 

^' A method according to any of the preceding claims, 

wherein the modified appearance is displayed in combination with or 
supported by further visual and/or auditive information. 
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7- A method according to claim 6, wherein data relating 

to the modified appearance are displayed as further information. 

^' A method according to claim 6 or 7, wherein moving 

images in the form of a video clip and the like are displayed as further 

information. 

9- A method according to any of the preceding claims, 
wherein said body part is a person's head, and the modified appearance 
comprises modifications In one or more aspects of the hairstyle, hair 
jewels, hats, the make-up, accessories in the form of glasses, coloured 
contact lenses, earrings, piercings and the like, and also modifications 
of the nose, mouth, lips, teeth, eyes, ears, eyebrows, moustache, beard 
and other facial features, among which the effect of a facelift or other 
reconstructive work, wherein for each aspect to be modified a separate 
virtual image data space is created or wherein aspects to be modified are 
comprised by a common virtual image data space. 

10- A method according to any of the preceding claims, 
wherein said second image data are selected and/or computed in dependence 
on said position data from virtual image data elements pre-stored in a 
data file. 

^1- A method according to any of the preceding claims, 

wherein said first and second image data are linked in such a manner that 
portions of said body part which will not be visible In the modified 
appearance are not shown, and that portions of said body part that will 
be visible are shown, by means of extrapolation, if desired, by adding 
to a respective virtual image data space respective virtual image data 
el ements . 

12- A method according to any of the preceding claims, 

wherein said virtual image data space is created relative to a generic 
type of said body part, wherein data representative of the dimensions 
and geometry of said body part are generated from said first image data, 
on the basis of which data said generic body type is scaled into 
commensuration with said dimensions and geometry of said body part 
according to said first image data. 

A method according to claim 11, wherein said second 
image data are scaled into commensuration with said scaled generic body 
type, and said virtual image data space is modified by said scaled second 
image data. . 
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^ '"^^'^od according to claim 12 or 13, wherein one or 
more portions of said first image data are substituted by image data 
derived from said scaled generic type of said body part. 

^ method according to any of the preceding claims 
wherein one or more aspects of said displayed modified appearance are 
interactively modified during display by modifying said second image data 
and/or said third image data. 

"'^^f'od according to any of the preceding claims 
wherein tracking means are coupled with said body part, and said body pari 
IS placed or held in a reference position for calibration purposes. 

^ according to any of the preceding claims 

wherein said image data and said image data elements Inclusive said virtual 
image data space and a generic body part type are three-dimensional digital 
representations. 

^ ""^t^^od according to any of the preceding claims 
dependent on claim 6, wherein said further information comprises further 
image data having the form of further virtual image data elements which 
are added to a virtual image data space. 

'"^^^'^ according to any of the preceding claims 
wherein said image data are provided and displayed as video data with a 
standard picture frequency of preferably 25 pictures/second or higher. 

^ ^^vice for displaying at least part of a person's 
body with a modified appearance following a method according to any of 
the preceding claims, said device comprising first means for providing 
first image data representative of the unmodified appearance of said body 
part, second means for providing second image data representative of a 
modification in the appearance of said body part, position determining 
means for providing position data representative of the position and 
spatial orientation of said body part, processing means for electronically 
processing said first and second image data in dependence on said position 
data, to produce third image data representative of said modified 
appearance of said body part, and means for displaying said third image 
data characterized in that said second means are arranged for providing 
second image data having the form of virtual image data elements, said 
processing means are arranged for creating a virtual image data 'space 
comprising an aspect of said body part to be modified, for adding to said 
virtual image data space virtual image data elements and for linking said 
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virtual image data space thus formed and said first image "data not 
comprised by said image data space to produce third image data, and for 
displaying same on said display means. 

A device according to claim 20, wherein said first means 
comprise recording means having at least one fixedly disposed video camera, 
and in that said tracking means comprise one or more trackers for 
determining the position and the spatial orientation of said body part 
relative to said recording means. 

^ device according to claim 21, further comprising a 
semi-transparent mirror, wherein said recording means are disposed on one 
side of said semi-transparent mirror, whilst said display means are 
disposed on its other side, in such a manner that said image displayed 
by the display means can be viewed from the said other side of said semi- 
transparent mirror as if said person is looking at said body part with 
its modified appearance in front of a mirror. 

device according to any of the claims 20 - 22, wherein 
said second means comprise storage means for storing second image data 
comprising one or more patterns or models composed of virtual image data 
elements representative of an intended modification in the appearance of 
a respective body part and/or further image data and control data. 

^ device according to claim 23, wherein said storage 
means comprise a read-only memory in the form of a smart card, a magnetic 
card or another information carrier for storing information in digital 
form thereon. 

^ ^^vice according to claim 23, wherein said storage 
means are disposed at a location remote from said device, said device and 
said storage means having transmission means for remote exchange of second 
image data and/or further image data and control data. 

A device according to claims 23, 24 or 25, wherein said 
second image data and/or said further image data in said storage means 
are linked to position data. 

A device according to any of the claims 23, 24 or 25, 
wherein said second image data and/or said further image data in said 
storage means are linked to data identifying a particular pattern or model 
of said image data. 

device according to any of the claims .20-27, wherein 
said first and second means and said processing means are arranged for 
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recording, storing and processing of three-dimensional digital image data, 

A device according to any of the claims 20 - 28, further 
comprising input means which are coupled to said processing means, inter 
alia for interactively changing one or more aspects of the modified 
appearance of said body part while it is being displayed. 

A device according to any of the claims 20 - 29, further 
comprising output means coupled to said processing means, in particular 
a printer for providing at least a print of said body part with its 
modified appearance. 

^ device according to any of the claims 20 - 30, further 
comprising means for presenting further visual and/or auditive information 
in combination with or in support of said display of said body part with 
its modified appearance. 

Storage means for use in combination with the device 
according to any one or •more of the claims 23 - 31. 

Storage means, comprising image data composed of virtual 
image data elements and/or further data. 

Storage means according to claim 33, wherein said image 
data and/or said further data are stored in said storage means associated 
with position data. 

Storage means according to claim 33, wherein said image 
data and/or said further data are stored in said storage means associated 
with data identifying a particular pattern or model of said image data. 
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